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(57) Abstract 

A system, method, and apparatus for enabling a user to draw high resolution graphical images onto a monitor or display screen are 
presented. Graphical location indicia corresponding to display screen locations are displayed on the monitor or display device. When the 
lip of a stylus pen is placed onto the monitor or display device, a location indicator displayed at display location thereunder is optically 
detected by the stylus pen. Tlie stylus pen then transmits corresponding electronic signals to a subsystem which determines a pixel location 
coiTcsponding to delected location indicator. A tracking algorithm is also presented wherein vector indicia are projected in a localized 
region surrounding the location of the stylus pen. The tracking algorithm pemiits fast tracking of the stylus pen at rates exceeding the 
refresh speed, as well as minimizing the obstruction to the graphical images displayed on the monitor or display device. 
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STYLUS PEN FOR WRITING ON THE MONITOR 

BACKGROUND OF THE PRESENT INVENTION 

Field of the Invention 

The present invention relates generally to computer 
graphics, and more particularly to inputting high 
5 resolution images into a computer system by moving a stylus 

pen over a display screen. 

Description of Related Art 

There has been a need for accurate computer pens, 
known as light pens or stylus pens, which would allow users 

10 to input high resolution images, such as handwriting, into 

computer systems. At present, slow, low resolution 
computer pens are available. Ideally, a stylus pen used in 
connection with a graphical user interface should make 
operating a computer system more intuitive and user- 

15 friendly. The goal of prior art .stylus pens was to allow 

a user to input a graphical image into a graphical user 
interface by placing the stylus pen over the display device 
and drawing the graphical image, similar to using a writing 
pen, but the prior art stylus pens could not keep pace with 

20 a user's movements were inaccurate, and provided poor 

graphic resolution . 

Stylus pens in general have a configuration similar to 
that of writing pens. They are used to effectively write 
on the computer monitor screen. For decades, engineers 

25 have tried to develop a display level input device, i.e., 

a location/touch sensitive display screen. The key factor 
concentrated on in prior art has been how to get the 
display screen to register the stylus location rather than 
have the stylus determine its location on the screen. A 

30 brief analysis of existing technology demonstrates that 

some success has been found by switching the direction of 
the stylus location determination. Specifically, the 
stylus pen can be used to determine its location on the 
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display surface, rather than trying to get the display 
surface to register the location of the stylus pen. To 
this extent, prior stylus pens rely on detecting the 
passage of the display monitor's electron beam as the 
5 display is refreshed. The timing of the electron beam 

detection is used to determine the stylus pen's display 
location . 

Prior stylus pens operate using a variety of 
techniques to produce a graphical effect on the display 

10 screen. Such techniques include or require (1) a specially 

designed display screen, and/or (2) the tracking of the 
vertical synchronization pulse, to name a few. 

U.S . Patent 3, 54 3, 2 40 issued to Miller, et . al, 
discloses a stylus pen operating with a remote display 

15 which includes a set of counrers. The counters are reset 

by computer instruction and count certain events. When the 
stylus pen detects a pass ing electron beam, an electronic 
pulse is produced by the stylus pen's circuitry causing the 
counters to stop. The values of the counters can then be 

20 used to determine the pixel location of the stylus pen. 

This technique has proven over time to be slow and 
inaccurate . 

U.S. Patent 3,949,391 issued to Benjamin, discloses a 
plasma panel stylus pen system, wherein the synchronization 

25 electron beam is first coarsely detected when the stylus 

pen touches the screen and thereafter more finely detected. 
The foregoing, however, requires a specialized display 
device having circuitry which only scans the area about the 
coarsely detected region to determine the location of the 

30 stylus pen. 

U.S. Patent 5,187,467 issued to Myers, discloses a 
stylus pen system capable of detecting horizontal and 
vertical synchronization signals . Here counters are 
utilized to determine where the stylus pen^s location is 

35 based on the detection of a synchronization pulse. 
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U.S. Patent 3,337,860 issued to O'Hara, U.S. Patent 
3,413,540 issued to Haring, U.S. Patent 5,227,622 issued to 
Suzuki, U.S. Patent 4,620,107 issued to Frame, and U.S. 
Patent 4,675,665 issued to Halliwell each tend to disclose 
systems in which the location of the stylus pen is 
determined by detection of the passing of and 
synchronization with the scanning electron beam as it moves 
across the monitor's screen. 

Each of the above stylus pen systems are 
disadvantageous because of reliance on detection of the 
passing of and synchronization with the monitor's scanning 
electron beam. This makes the system inherently slow and 
inaccurate due to the difficulty in catching and 
synchronization with a single pulse of the passing electron 
beam. Furthermore, the prior art stylus pen systems have 
a wide field of view that encompasses several horizontal 
lines of the electron beam raster. Therefore, the detected 
pulse may actually be a series of discontinuous pulses or 
may have an inaccuracy of two to three raster lines. 
Additionally, the size of the field of view varies 
depending on the type of phosphor used within the display 
screen, the distance between the pen and the phosphor, and 
the intensity of the electron beam's light. Furthermore, 
these prior art systems generally do not work with liquid 
crystal displays (LCDs) or plasma displays common to laptop 
computer systems. Reliance on the passing electron beam 
limits the pixel location sampling rate of the stylus pen 
to the screen refresh rate. Additionally, detection of the 
passing electron beam is more suited for monochrome CRT 
displays as opposed to the present day more common, color 
displays . 

Therefore, there is a need for a method and system 
which determines the location of a stylus pen independently 
from the electron beam and which does not require special 
circuitry built into the display screen's surface. There 
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is also a need -to provide a system wherein a conventional 
display devige and a general purpose computer system can be 
easily adapted for use with the stylus pen. There is also 
a need to efficiently track the movement of the stylus pen 
to produce a high resolution mechanism for producing 
graphics on a display screen via a stylus pen. 

SUMMARY OF THE INVENTION 

An embodiment of the present invention provides a 
stylus pen that can be used to effectively produce graphics 
on a computer's display when the stylus pen is touched to 
the display screen . Such graphics are produced via the 
stylus pen reading a "hidden" or inlayed pixel pattern 
embedded in the display screen's picture or graphics. 

An embodiment of the present invention determines the 
location of a stylus pen independently of a monitor's 
scanning electron beam. 

An embodiment of the present invention provides a 
system wherein a conventional display device and a general 
purpose computer system can be easily adapted for use with 
0 the stylus pen . 

An embodiment of the present invention efficiently 
tracks movement of the stylus pen at a rate that can keep 
pace with a user's normal hand movements. 

An embodiment of the present invention displays 
5 location indicia or an inlayed pixel pattern on the display 

device's screen. The location indicia are readable by the 
stylus pen. In one embodiment, the location indicia is 
provided by a subsystem, or circuitry which can be 
connected between the display device and a general purpose 
0 computer system or built into the general purpose computer. 

The subsystem receives graphic images for the display 
device and may interleave the graphic images with the 
location indicia. The graphic images, interleaved with the 
location indicia are then displayed on the display device. 
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When the stylus pen is directly over an area on the display 
device, the stylus pen reads the interleaved pixel location 
indicia, and provides the location indicia to the 
subsystem. The subsystem then determines the location 

5 and/or movement of the stylus pen. The determined location 

of the stylus pen is then provided to the general purpose 
computer system, via a mouse port or other suitable 
mechanism or data port. 

In an alternate embodiment, a computer system with the 

0 stylus pen is disclosed without a subsystem externally 

connected between the stylus pen and the general purpose 
computer. The computer system transmits graphic images 
interleaved with the location indicia to the display 
device. The stylus pen can read the location indicia from 

5 the screen and transmit the read location indicia from the 

screen to the computer system. The computer system then 
calculates the location and/or movement of the stylus pen- 
Additionally, a method for locating and tracking the 
movement of the stylus pen is presented. The status of the 

0 stylus pen is continuously polled. When the status of the 

stylus pen changes from disengaged to engaged (i.e., the 
stylus pen is placed in contact with the display device's 
screen) , the location indicia is displayed in an array 
format of the display device. The stylus pen can optically 

5 read the location indicia. The read pixel location indicia 

is used to determine the location of the stylus pen. After 
determining the location of the stylus pen, a direction box 
including localized vector indicia about a smaller 
localized area surrounding the determined stylus pen 

0 contact location is displayed on the display device. As 

the stylus pen is moved across the screen, the localized 
vector indicia is read, movement vectors describing the 
movement of the stylus pen are determined, and the 
direction box is repositioned to follow the movement of the 

5 stylus pen. Pen-like graphics can be placed on the screen 
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to provide high resolution "wriiiing" on the screen as the 
stylus moves "across the screen. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Various objects and advantages of this invention will 
5 become apparent and more readily appreciated from the 

following description of the presently preferred exemplary 
embodiments, taken in conjunction with the accompanying 
drawings, of which: 

FIGURE 1 is a block diagram of a system 
10 configured in accordance with an exemplary embodiment of 

the present invention; 

FIGURE 2 is a block diagram of an exemplary 
stylus pen in accordance with the present invention; 

FIGURE 3 is a diagram of a monitor displaying 
15 pixel location indicia in accordance with the present 

invention; 

FIGURE 4 is a diagram of a monitor displaying 
localized vector indicia about a location of a stylus pen 
in accordance with the present invention; 
20 FIGURE 5 is a block diagram of an exemplary 

subsystem in accordance the principals of the present 
invention; 

FIGURE 6 is a flow diagram describing the 
operation of the subsystem in accordance with the present 
25 invention; and 

FIGURE 7 is a block diagram of a computer system 
in accordance with another exemplary embodiment of the 
present invention . 



6 



BNSDOCID: <WO_0070551 Al_L> 



wo 00/70551 



PCTAJSOO/12995 



DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
OF THE PRESENT INVENTION 

An exemplary stylus pen in accordance with the present 
invention implements a digital solution for whiteboard or 
5 freehand drawing technology which allows the stylus pen to 

be used for a wide variety of uses. The stylus pen can be 
used to allow freehand input directly into graphical 
operating environments and applications during business 
conferences. Graphic artists can use an exemplary stylus 

10 pen apparatus in a freehand manner directly with drawing 

applications. Persons with carpal tunnel syndrome, or who 
are otherwise unable to type, may use the exemplary stylus 
pen instead of a keyboard. 

By designing the exemplary stylus pen to operate as a 

15 peripheral, the stylus pen can be flexibly used with 

general purpose computer systems. The exemplary peripheral 
design permits the stylus pen to be used with a wide 
variety of commercially available monitors. The stylus pen 
can also be used with television sets configured to operate 

20 as a computer display, as well as backlit monitors and 

plasma panels. The exemplary peripheral design also 
permits the stylus pen to be used either in conjunction 
with a mouse or as a substitute therefore. The foregoing 
may be achieved by integrating the exemplary stylus pen 

25 with a device enabling the monitor to act as a transmitter. 

An optical sensor in the stylus pen acts as a receiver. 

Referring now to FIGURE 1, there is illustrated a 
block diagram of an exemplary computer system, referred to 
generally by the numeral 100, configured in accordance with 

30 a first embodiment of the present invention. The system 

100 includes a general purpose computer system 105 with a 
mouse port 110, a keyboard port 115, and a monitor port 
120. The computer system 105 performs general computer 
processing tasks and can be operated in a stand-alone mode 

35 or be connected to additional computers to form a computer 
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network. The general purpose computer 105 receives user 
inputs through the mouse port 110 and the keyboard port 115 
and transmits outputs from the monitor port 120. 

The computer system 105 is configured to receive 
5 inputs, and to transmit outputs using electronic data 

signals. Peripheral devices are connectable to the 
computer system 105 to serve as an interface for inputs to 
be received from, and outputs to be communicated to users. 
Peripheral devices include, for example, a keyboard 125, a 
0 mouse 130, and a monitor 135. The keyboard port 115 is 

connectable to keyboard 125, permitting the user to provide 
text-based inputs. The mouse port 110 is connectable to a 
motion based user input device, commonly known as a mouse 
130 (although not shown connected) . 
15 The monitor port 120 is connectable to a monitor 135, 

wherein outputs are graphically displayed to the user via 
a display screen 136. The monitor 135 is also used to 
assist the user in providing inputs, by displaying 
graphical user interfaces (GUI) , A graphical user 
20 interface prompts the user to provide inputs to the 

computer system 100 in a user-friendly, intuitive manner. 
For example, a GUI is often used to display a number of 
items associated with various user inputs. The user 
provides an input by pointing to the item associated with 
25 the input and pressing a button on the mouse 130. The 

foregoing process is known as pointing and clicking. 

Another example of a peripheral device is a stylus pen 
140. The stylus pen 140 allows the user to input high 
resolution graphical images into a GUI by placing the 
30 stylus pen 140 over the monitor 135 and over a particular 

monitor displayed pixel (s), and drawing a graphical image 
directly (seemingly) on the screen. The stylus pen 140 
itself, however, does not directly write to the monitor 
135. When the stylus pen is placed against the monitor's 
35 display screen 136, the stylus pen 140 detects a location 

8 
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indicator in the form of an illuminated pixel. The 
location indicator is found within an array of location 
indicia that are optically displayed at the particular 
pixel location (s) on the display screen 136. The 
5 recognized location indicator can be referred to as the 

pixel location of the stylus pen. Thus, the pixel location 
of the stylus pen 140 is ascertained and provided to the 
computer system 105 from the detected location indicator. 
The computer system 105 can then provide the pixel location 

10 of the stylus pen 140 to various software programs. 

Virtually everything displayed on the screen 136 of 
conventional monitors, such as monitor 135, must be 
provided by an external monitor driver source due to 
conventional monitors being passive devices. The computer 

15 system 105 provides a stream of graphical image data for 

display on the monitor 135. The aforementioned location 
indicia must also be provided to the monitor 135. In the 
exemplary embodiment shown in FIGURE 1, the aforementioned 
location indicia is provided by a subsystem 145, The 

20 subsystem 145 is connected to the monitor 135 and outputs 

the location indicia thereto. 

Referring now to FIGURE 2, there is illustrated a 
block diagram of an exemplary stylus pen 140 in accordance 
with the present invention. The stylus pen is placed in 

25 contact with the monitor 135 so that it can optically view 

the pixels on the screen 136. When the location indicia is 
displayed on the monitor 135, the location indicia 
associated with the particular pixel over which the stylus 
pen 140 is located is detected by an optical sensor in the 

30 stylus pen 140. The optical sensor converts the location 

indicia to a data signal and transmits (via a direct 
connection or a cordless interface) the data signal to the 
subsystem 145. The subsystem 145 can then determine the 
pixel location of the stylus pen 140. 
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The stylus pen 140 includes a fiber optic bundle 205 
mounted into a swivel tip 210. The fiber optic bundle 205 
provides an optical route for received light to an optical 
sensor 215. The tip of the fiber optic bundle 205 is 
5 mounted such that it has a flat surface 225 that stays 

flush with the display screen via the swivel tip 210. The 
coiT±)ination of fiber optic bundles 205, the flat face 225 
and the swivel tip 210 allows the optical sensor 215 to 
optically read the location indicator displayed at a 
10 particular pixel under the tip of the fiber optic bundle ; 

205 with maximum clarity from most writing angles. 

The stylus pen 140 also includes a micro-switch 230 
disposed in a manner such that when the tip of the fiber 
optic bundle 205 is placed in contact with the monitor 135, 
15 the mechanical pressure from the contact causes the micro- 

switch 230 to be depressed. Depressing the micro-switch 
230 causes the stylus pen 140 to signal the subsystem 140 
that the stylus pen 140 is in an engaged mode. 
Additionally, where the fiber optic bundle 105 is moved 
20 away from the monitor 135, the stylus pen 140 signals the 

subsystem 145 that stylus pen is in a disengaged mode. 

In another embodiment of the present invention, a 
light intensity is calculated using the optical sensor 215 
instead of using the micro-switch 230. By utilizing a 
25 light intensity reading to place stylus pen in engage or 

disengage mode, a user need only place the tip of the 
stylus pen on the screen to engage it. Furthermore, by 
eliminating the micro-switch 230, the number of circuits in 
the stylus pen 14 0 is decreased and the micro-switch will 
30 not become a critical failure point in the system due to 

wear. Furthermore, by utilizing light intensity, the user 
can accomplish a substantially similar result by touching 
the monitor screen or having the tip of the stylus pen 
close to the monitor screen. 

10 
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When the fiber optic bundle 205 is placed in contact 
with the screen 136 of the monitor 135 directly over a 
location pixel, the optical sensor 210 detects the light 
wavelength (or color) of the light radiated from the pixel, 

5 and transmits a corresponding electronic signal to the 

subsystem 145. The pixel location of the stylus pen 140 is 
then determined by the subsystem 145 by ascertaining the 
pixel location associated with the color corresponding to 
the electronic signal. The stylus pen 140 also includes 

0 two buttons 240a, 240b/ which correspond to the right and 

left buttons, respectively as found on a standard two- 
button mouse. 

The full matrix mode will now be described. When the 
stylus pen 140 first makes contact with the screen 136 of 
5 the monitor 135, the graphical image displayed on the 

screen 136 is replaced by full matrix mode location indicia 
detectable by the stylus pen 140 and identifiable by a 
location determination algorithm which will be described 
below. 

0 Referring now to FIGURE 3, there is illustrated a 

diagram of the monitor screen 136 displaying the location 
indicia during the full matrix mode. The image displayed 
on the monitor screen 136 is composed of a two-dimensional 
array of pixels 305. Each pixel 305 displays a 

5 predetermined color for a predetermined time period. The 

time period can be equal to or related to the refresh 
period of the monitor 135. 

Each area 305 of the display screen 136 is defined by 
its own color pattern. An area may be a group of pixels or 

0 a single screen pixel. In full matrix mode the display 

screen 136 is defined by a fixed number of rows 315 and 
columns 310. Each row 315 and column 310 is then "labeled" 
with its own unique color. An equation is created to 
define a color to fill each area or pixel 305. The color 

5 is from either the row 315 or column in which the area 305 
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resides. An exemplary equation for an 800 x 600 resolution 
display screen using 100 colors for columns is now 
described: 

Define the colors for the columns 310 (Xcolor) and 

5 colors for the rows 315 (Ycolor) in a sequential fasion. 

Note that in this example the height of the display screen 

is 0.75 times the width of the display screen, thus there 

will be three quarters the number of colors used for rows 

as for columns. The number of colors for columns (Ncolors) 

10 is used to calculate a color pattern. 

Thus, Xcolor = ( 0 to 99) 
Ycolor = (100 to 174) 

The total number of pixels = pixel resolution for che 
rows on the display. 

15 Example: For 800 x 600 resolution display 

Total pixels = 800 

Next, a color for each pixel column 310 is calculated. The 

result will be an integer value that can be used to cross 

reference a color table: 

20 Xcolor = [ {Pixel (X) /Total Pixels) x Ncolors] Truncated 

Example : Xcolor (X=3C0) =[( 300/800) x 100] Truncated 

Next, a color for each pixel row is calculated. The result 

will be an integer value that can be used to cross 

reference a color table: 

25 YColor= [ (Pixel (Y) /Total Pixels) x Ncolors] Truncated 

+ Ncolors 

Example : Ycolor (Y-233) = [ (233/800) xlOO] Truncated 
+ 100 = 129 

Next, it is decided which color is used for each pixel 305: 

30 If Pixel (X) + Pixel (Y) is odd, then fill the 

pixel with Xcolor for that pixel location. 
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If Pixel (X) + Pixel (Y) is even, then fill the 
pixel with Ycolor for that pixel location. 

Example : For Pixel (X=300, Y=233) fill with Xcolor, 
which is color #37 

To determine the screen location of the stylu.s when it 
is first touched to the screen, the location indicia is 
flashed on the display screen so that the optics of the 
stylus can read the color (Xcolor or Ycolor) of the 
location. The stylus, via the optical sensor, is polled so 
that the detected color can be determined. The color is 
represented as the Xcolor number so that, via the above 
equations, the location of the stylus can be determined. 
The equations that follow disclose an exemplary technique 
for determining the location of the stylus based on the 
color read by the stylus optics. 

First, cross reference the selected color table to 

determine Xcolor and Ycolor. (Note: If a color is detected 

that is not in the selected color table,, the system will 

log an error) : 

If the color returned is between 0 and Ncolors, 
then it is Xcolor. 

If the color returned is between (Ncolors+1) 
and (1 . 74*Ncolors) , then it is Ycolor 

Example : Colors returned from sensor: Color 
#129 and Color #37 

Therefore Xcolor=37 and Ycolor=129 
Reverse calculate the pixel reference position for the 
pixel column from Xcolor: 

ReferencePixell {X)=[Xcolor* (TotalPixels) / (Ncolors) ] 
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Example : Xcolor = 37 

ReferencePixel = [37*800/100] = 296 

Reverse calculate the pixel reference position for the pixel 

row from Ycolor: 

5 ReferencePixel (Y) = [ Ycolor-Ncolors) * (Total 

Pixels) / (Ncolors) ] 

Example : Ycolor = 129 

Reference Pixel=[ (129-100) *800/100] =232 

In the example shown, the pixel position read was (300 
10 X 233), and the reference position returned was (296 x 232) . 

This is extremely close to the actual stylus position. The 
series of equations and calculations shown are capable of 
reaching double pixel resolution through the simple use of 
more colors. 

15 Once the stylus location on the display screen is 

determined, then a moving overlay pattern is positioned on 
the screen about the stylus pen's location. The moving 
overlay can be substantially any shape, but preferably is 
square. Once the moving overlay pattern is displayed on the 

20 display screen tracking mode is initiated. 

Referring to FIGURE 4, the tracking mode will now be 
described. In order to track movement of the stylus across 
the screen, tracking mode is used. The computer system 
utilizes the tracking mode at a speed which allows enough 

25 stylus pen positions to be gathered for effective use in a 

writing or drawing application- 
Tracking mode is used to track the stylus pen 140 as it 
is being moved across the screen 136 from the initial point 
of contact with the screen 136 to other positions on the 

30 screen. An overlay pattern 405 (exploded view) is displayed 

on top of the graphic images 415 displayed on the screen 136. 
The. overlay pattern 405 must be both visible to the user and 
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allow the user to see the graphic images 415 that are being 
* drawn on' by the stylus 140. 

An exemplary overlay pattern 405 is 20x20 pixels but 
could be larger than 100 x 100 pixels in size. The overlay 
5 pattern 405 comprises vector indicators 404 in every other 

pixel location within the overlay pattern 405. The vector 
indicators 404 are each assigned a different color and may 
create a checkerboard pattern over graphic image 415. The 
graphic image portion that is underneath the overlay pattern 

10 405 is converted to a black and white (gray scale) so that 

the graphic image portions pixels are not confused with the 
vector indicators 404. Note that the pattern does not have 
to be checkerboard but could be octagons, pentagons, 
triangles or substantially any repeating or interlocking 

15 pattern. It is further noted that the patterns could 

incorporate as few as one-third to one-eighth of all the 
pixels in the overlay pattern, but preferably one-half the 
pixels provides the greatest stylus tracking accuracy and 
resolution at normal stylus movement speeds, 

20 In other words, the portion of the original image that 

is underneath the overlay pattern 405 is converted to gray 
scale, while a checkerboard pattern of half the pixels in the 
overlay pattern are each displayed as a different color. 

In FIGURE 4, each numbered (or Xed) location in the 

25 overlay pattern 405 represents a uniquely colored vector 

indicator 404. The center location represents the known 
stylus pen position 410. The stylus pen position 410 is 
centered in the overlay pattern 405 on refresh of the screen 
136. 

30 As the stylus pen 140 is moved across and in contact 

with the screen, the stylus pen 140, via the optical sensor, 
reads a localized indicator 404 that is displayed under the 
stylus pen's location. The color of the localized indicator 
4 04 is determined thereby providing information to the 
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computer system which can be used to determine new stylus pen 
position information and pen movement vector information. 

Upon a screen refresh, the moving overlay pattern is 
recentered about the last determined position of the stylus 
5 pen. Furthermore, a graphic trail following the stylus can 

be created as the pen moves across the screen. The graphic 
trail simulates writing on the display screen 136. In a 
preferred embodiment, the stylus pen's optics can read and 
interpret the vector indicators 404 at a rate that exceeds 
10 the refresh rate of the screen. Thus, the graphic trail 

behind the pen can be updated with a plurality of stylus pen 
locations on a screen refresh. Furthermore, the trail behind 
the stylus pen (see FIGURES 1 and 7) can be a high resolution 
trail of one or more pixels in width. 
15 The overlay pattern 405 is preferably large enough so 

that the stylus pen 140 is not likely to be moved outside of 
the overlay pattern 405 by a user for the duration of a 
refresh period. The preferred embodiment can sense three to 
five or more vector indicators per refresh cycle, which 
20 allows the exemplary embodiment to take 300 to 500 or more 

stylus location reads per second. Since one exemplary 
application of the present exemplary invention is for 
"brainstorming tools" like white boards, or for artistic 
tools like freehand drawing, it is important to allow a user 
25 to move the stylus pen on the screen at speeds that are 

comfortable to the user and at speeds which will allow the 
user to view an accurate depiction of what they are drawing 
on the screen instantly. 

The stylus pen can be tracked anywhere on the screen 
30 with minimal obstruction to graphical images being displayed 

on the screen. The location of the pen is determined at a 
rate that far exceeds the refresh rate of the screen. That 
is, the pen location is determined at a rate that is three 
to five times more often than the refresh rate. The speed 
35 of stylus pen location calculations is limited by the speed 

16 



BNSDOCID: <WO_0070551At_l_> 



wo 00/70551 



PCT/USOO/12995 



of the host computer system 105 and/or the integrated circuit 
speeds found in the computer's graphic circuitry, not the 
speed of the screen refresh rate. 

Referring back to FIGURE 2, the optical sensor 215 in 

5 the stylus pen must be able to discriminate discrete colors 

in a large color spectrum produced by a color display 
monitor. An exemplary computer system utilizing 8 bit (256 
color) technology is sufficient to meet the design 
requirements for matrix mode and tracking mode. CCD or CMOS 

0 optical sensors have been determined to be able to provide 

the required optical sensing capability in an exemplary 
stylus pen. It is understood that other types of optical 
sensors could be used without departing from the spirit of 
the invention. Furthermore, it is understood that CCD and 

5 CMOS sensors could also be utilized with color technology at 

10 bit (1024 colors) or 24 + bit (true color) . As the color 
technology advances and pixel size decreases, the vector 
indicators may each comprise a plurality of pixels instead 
of a single pixel each. 

0 Referring now to FIGURE 5, there is illustrated a block 

diagram of the subsystem 145 in accordance with an embodiment 
of the present invention. The subsystem 145 includes a 
central processor 505 which executes instructions stored in 
non-volatile memory 510, such as Programmable Read-Only 

5 Memory (PROM) . The central processor 505 receives inputs 

from the stylus pen 140, the mouse 130, and graphical images 
from the monitor port 110 of the computer system 105, and 
provides outputs to the monitor 135 and the mouse port 110 
of the computer system 105. 

0 The central processor 505 receives the input from the 

stylus pen 140 via the stylus port 525. The inputs received 
from the stylus pen 140 include a status signal describing 
the mode of the stylus pen, the electronic signal provided 
by the optical sensor 215, and signals corresponding to the 

5 buttons 240a, 240b. When the stylus input port 525 receives 
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the engaged signal from the stylus pen 140, the central 
processor 505 retrieves the electronic signal provided by the 
optical sensor 215 through the stylus port 525. The central 
processor 505 then determines the pixel location and/ or 
5 offset vector based on the signal provided from the stylus 

pen's 140 optical interpretation of the location indicator's 
color or the localized vector indicia's color. 

The pixel location and/or vector indicator location of 
the stylus pen 14 0 can be determined in a number of ways that 

10 would be understood by one skilled in the art . In one 

scheme, the electronic signal is compared to determine the 
associated vector indicator location within the overlay 
pattern. The indicator location is then offset by the center 
of overlay patterns location. The non-volatile memory 510 

15 can be rewritten or reinitialized so that on initialization 

a sampling of the screen's color variation capabilities can 
be loaded or reloaded into the look-up table. 

The location of the stylus pen 140 may be determined by 
transmitting the location or vector indicator information 

2 0 from the subsystem to the mouse port 110 of the general 

purpose computer system 105. In order to provide inputs to 
the computer 105 from the subsystem 145, a mouse/stylus 
driver 530 is placed between the central processor 505 and 
the mouse port 110. The mouse/stylus driver 530 enables the 

25 system to convert an absolute location to a mouse motion 

signal that is transmitted to the mouse port 110 of the 
computer system 105. A user may use the computer system 

105 to perform functions with or without using the stylus pen 
140. If the stylus pen 140 is not being used, the user may 

30 use the mouse 130 instead. In order to use the mouse 130 

with the exemplary computer system 105, the subsystem 145 is 
connected to and receives signals from the mouse 130 via the 
auxiliary mouse port 535. The mouse driver 530 includes 
decision logic that can selectively pass the signals received 

35 from the mouse 130 or the stylus pen 140 to the mouse port 
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120. When the stylus pen 140 is disengaged, the inputs of 
the auxiliary 

mouse port 535 are provided to the computer system 105. When 
the stylus pen 140 is engaged, the mouse 130 is disengaged 
5 and 

information data which will move the cursor to the detected 
location of the stylus pen is provided to the computer system 
105. The mouse driver 530 will determine that the stylus pen 
is engaged when the micro-switch 230 is depressed. 

0 The central processor's 505 control of the monitor 135 

is effectuated by a screen driver 540. The screen driver 540 
is based upon a standard screen driver as can be found in 
general purpose computer systems 105. The screen driver 540 
allows the subsystem 145 to interrupt the graphical images 

5 for display on the monitor 135 and overlay it with the 

location indicia or the overlay pattern comprising vector 
indicators that are detectable by the stylus pen 140. The 
central processor 505 outputs the images to be displayed to 
the screen driver 540. The screen driver 540 then causes the 

0 monitor 135 to display the prescribed images. 

The screen driver 540 can be provided with preferably 
one of three patterns by the central processor 505 - (1) 
unmodified graphical images from the computer system 105; (2) 
the location indicia (full matrix mode); or (3) the moving 

5 overlay pattern interposed with or overlaying the graphical 

images received from the computer system 105. The graphical 
images are received from the computer system 105 via a screen 
input port 545. The screen input port 545 is connectable to 
the display port 120 of the computer system 105. When the 

0 central processor 505 displays the graphical images from the 

computer system 105 for display, the central processor 505 
outputs the contents of the screen input port 545 to the 
screen driver 540. 

The location indicia are stored in the non-volatile 

5 memory 510. When the central processor 505 selects the 
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location indicia for display (due to the stylus pen 140 being 
engaged) on the monitor 135, the central processor 505 
accesses the location indicia from the non-volatile memory 
510 and outputs the pixel location indicia to the screen 
driver 540 such that it is interposed with or overlays the 
image already displayed on the screen. The localized vector 
indicators are also stored in the non-volatile memory 510. 
When the central processor 505 selects the moving overlay 
pattern for display (tracking mode) , the localized vector 
indicators are interposed with the graphical images provided 
by the computer system 105 in an area about the stylus pen's 
position. The central processor provides the graphical images 
from the screen input port 545 and accesses the vector 
indicia from the non-volatile memory 510. The localized 
vector indicators are interposed with an overlay of the 
graphical images about the pixel location of the stylus pen 
140 at the most recent refresh by the central processor 505 
and output to the screen driver 540. Note, that the stylus 
pen 140 can read the location indicia or localized vector 
indicators at a rate that exceeds the screen's refresh rate. 

Referring now to FIGURE 6, there is illustrated a flow 
diagram describing exemplary operation of the subsystem in 
accordance with the present invention . The subsystem 
constantly polls (step 605) the status of the stylus pen to 
determine whether the stylus pen is engaged (step 610) . When 
the stylus pen is not engaged, i.e. disengaged during step 
610, the subsystem forwards the graphical images received 
from the computer system to the monitor (step 615) . While 
the stylus pen is in a disengaged mode, steps 605 through 615 
are repeated and the stream of graphical images received from 
the computer system are displayed on the monitor. 
Furthermore, the mouse 130 is fully operational when the 
stylus pen 140 is disengaged. 

However, when the stylus pen becomes engaged during step 
610, the subsystem temporarily halts displaying the graphical 
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images received from the compurer system and instead displays 
the pixel location .indicia of the matrix mode (step 62Q) . 
The location indicia are displayed until the subsystem 
receives a signal from the stylus pen detecting the location 
5 indicia at the pixel location of the stylus pen (step 625) . 

In one exemplary embodiment, the location indicia may be 
interposed with the graphical images received from the 
computer system. When the subsystem receives a detection 
signal from the stylus pen identifying one of the location 
I 10 indicia for the pixel location of the stylus pen, the 

subsystem determines the pixel location of the stylus pen 
(step 630) and forwards it to the computer system (step 635) . 
The pixel location of the stylus pen 140 is communicated to 
the computer system 105 by calculating a vector from the last 
15 known mouse location to the pixel location of the stylus pen 

140. 

After determining the initial pixel location of the 
stylus pen during step 630, the pen can be moved across the 
display screen. The location of the stylus pen can be 

20 tracked by displaying a localized overlay pattern about the 

initial location of the stylus pen. At step 640, the 
graphical image received from the computer system is 
interposed or overlayed wirh the overlay pattern which 
comprises vector indicators. The overlay pattern is 

25 preferably centered about the pixel location of the stylus 

pen and displayed on the monitor. The subsystem then waits 
" until a 'next' detection signal is received from the stylus 

pen (step 650) . When the next detection signal is received 
from the stylus pen, the subsystem determines a two 

30 dimensional offset vector from the detection signal (step 

655) and forwards the two dimensional offset vector to the 
computer system (step 660) via the mouse port. The subsystem 
then updates the pixel location of the stylus pen (step 665) 
by adding the offset vector determined during step 655 to the 

35 last pixel location of the stylus pen. It is noted that a 
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plurality of detection signals may have been received and 
recorded between each screen refresh. At step 670, the 
subsystem polls the status of the stylus pen to determine if 
the stylus pen is- still engaged (step 675) , If the stylus 
5 pen is engaged, a determination is made as to whether the 

refresh period has expired (step 680) . If the refresh period 
has expired, the moving overlay pattern is interposed or 
overlayed with the graphical image received from the computer 
system 105 and positioned about the pixel location of the 
10 stylus pen 140 determined during step 665. Steps 650 to 680 

are repeated until the stylus pen becomes disengaged during 
step 675- When the stylus pen becomes disengaged during step 
675, the subsystem returns to the disengaged r?.cdc at step 
605. 

15 The foregoing exemplary system and associated exemplary 

methods advantageously permits a user to easily adapt a 
general purpose computer system 105 and conventional monitor 
135 to practice the present invention by simply connecting 
the subsystem 145 as shown in FIGURE 1. However, recent 

20 advances in microelectronics technology have provided for 

portable computers known in the art as laptop computers. A 
laptop computer usually includes a computer system integrated 
with a display screen. However, because the display screen 
is integrated with the computer system, a laptop computer is 

25 not as easily adaptable for use with the exemplary subsystem 

145 . Furthermore, the configuration and communication 
protocols of the display screen on many laptop computers are 
proprietary, as opposed to standardized, adding further 
complexity - 

30 Referring now to FIGURE 7, there is illustrated a 

computer system 705 configured in accordance with another 
embodiment of the present invention. The computer system 705 
includes a stylus pen port 710 for receiving signals from a 
connectable stylus pen 140, an integrated display device 715, 

35 and a central processing unit 720 for performing processing 
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functions. The stylus pen port 710 can be a dedicated port 
created by the computer manufacturer or a standard serial 
port. It is understood that the integrated display device 
can be substantially any type of color display device. The 
5 signals received by the stylus pen port 710 are forwarded to 

the central processing unit 720. The central processing unit 
720 responds to the inputs received from the stylus pen port 
710 by causing location indicia 305 and/or movable overlay 
patterns 405 to be displayed onto the integrated display 
W 10 device 715 in accordance with the flow chart of FIGURE 6 by 

executing sets of computer readable instructions. The sets 
of instructions can be stored in, for example, RAM 725, ROM, 
or any data storage or memory device. Until required by the 
central processing unit 720, the set of instructions may be 
15 stored in a variety of computer readable memory devices, for 

example in a hard disc drive 730, or in removable memory such 
as an optical disc 735 for eventual use in an optical disc 
drive 740, or a floppy disc 745 for eventual use in a floppy 
disc drive 750 . 

20 Although the invention has been described with a certain 

degree of particularity, it should be recognized that 
elements thereof may be altered by persons skilled in the art 
without departing from the spirit and scope of the invention. 
Therefore, the invention is limited only by the following 

25 claims and their equivalents. 
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WHAT IS CLAIMED IS: 

1. A computer system that allows a user to effectively 
write on a display screen with a stylus, said computer system 
comprising : 

a general purpose computer; 
5 a subsystem connected to said general purpose computer; 

a display screen for receiving graphics data from said 
general purpose computer and for displaying said graphics on 
said display screen, said display screen further receiving 
location indicators from said subsystem for display on said 
10 display screen; and 

a stylus adapted to optically read at least one said 
location indicator displayed on said display screen, said 
optically read location indicator being provided to said 
general purpose computer in the form of data so that a 
15 location of said stylus pen on said display screen can be 

calculated. 

2. The computer system of claim 1, wherein said 
location indicators are predetermined colored locations 
displayed on said display screen. 

20 3. The computer system of claim 1, wherein said 

location indicators are displayed at predetermined colored 
pixel locations on said display screen. 

4. The computer system of claim 1, wherein said stylus 
optically reads a plurality of location indicators during 

25 single refresh period of said display screen. 

5. The computer system of claim 1, further comprising 
a peripheral point and click device that provides point and 
click information to said personal computer. 
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6. The computer system of claim 5, wherein said point 
and click device is inoperative when said stylus is reading 
said location indicators on said display screen. 

7, A method of determining a position of a stylus on 
5 a display screen comprising the steps of: 

touching a stylus to said display screen; 

displaying an array of color coded locations on said 
display screen; 

optically reading a color coded location, of said array 
10 of color coded locations, that is located under said stylus; 

and 

determining a first location of the stylus on said 
display screen based on the optically read color coded 
location . 



15 8. The method of claim 1, wherein each said color 

coded location is a pixel location on said display screen. 

9. The method of claim 1, wherein after the step of 
determining, the method further comprises the steps of: 

positioning an overlay pattern on said display screen 
20 at said first location such that said overlay pattern is 

substantially centered about said first location; 

moving said stylus across said display screen; 

optically reading a plurality of color coded vector 
locations within said overlay pattern; 
25 determining the locations of said moving stylus pen 

based on said optically read color coded vector locations; 
and 

repositioning said overlay pattern at one of said 
plurality of said optically read color coded vector 
30 locations. 
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10 . The method of claim 9 , wherein is said step of 
repositioning is performed once every screen refresh cycle. 

11. The method of claim 9, further comprising a step 
of displaying a trail effectively behind said stylus, said 

5 trail comprising at least said optically read color coded 

vector locations. 

12. The method of claim 9, wherein said step of 
determining said locations further comprises the step of 
substantially calculating a movement speed of said stylus. 

10 13. The method of claim 9, wherein said step cf 

determining said locations further comprises the step of 
substantially calculating a movement vector of said stylus. 

14. An apparatus for determining positions of a stylus 
on a display screen, said apparatus comprising : 
15 a stylus having an optical sensor for reading and 

discriminating between colors displayed on a display screen; 

an article of manufacture for implementing a stylus 
tracking procedure, said article of manufacture comprising: 
at least one computer readable medium; 
20 processor instructions contained on said at least 

one computer readable medium, said processor instructions 
configured to be readable by said at least one processor and 
thereby cause said at least one processor to operate to: 
poll a status of said stylus to determine whether said stylus 
25 is touching said display screen; 

display an array of colors on said display screen; 
receive an optically interpreted first detection 
signal from said stylus ; and 

determine the position of said stylus on said 
30 display screen based on said first detection signal. 
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15. The apparatus for determining positions of a stylus 
on a display screen of claim 14, wherein said processor 
instructions further cause said at least one processor to 
operate to: 

5 overlay an overlay pattern substantially about said 

determined positions of said stylus, said overlay pattern 
comprising a plurality of vector indicators; 

receive optically interpreted vector indicator 
signals from said stylus as said stylus is moved across said 
^ 10 display screens- 

tracking a plurality of stylus positions based on 
said received optically interpreted vector indicators; and 

positioning a new overlay pattern on said display 
screen about determined stylus positions. 

15 16. The apparatus for determining positions of a stylus 

on a display screen of claim 14, wherein said array of colors 
is display such that a plurality of the pixels of said 
display screen are different colors arranged in said array 
of colors. 

20 17- The apparatus for determining positions of a stylus 

on a display screen of claim 15, wherein said overlay pattern 
is smaller than 100 pixels by 100 pixels. 

18. The apparatus for determining positions of a stylus 
^ on a display screen of claim 15, wherein said overlay pattern 

25 comprises a plurality of colors positioned in predetermined 

locations about said overlay pattern. 

19. The apparatus for determining positions of a stylus 
on a display screen of claim 18, wherein said plurality of 
colors are positioned in a checkerboard pattern. 
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20. The apparatus for determining positions of a stylus 
on a display screen of claim 15, wherein said new overlay 
patterned is positioned on said display screen at a stylus 
position determined once every screen refresh cycle. 

5 21. A method for inputting a high resolution image into 

a personal computer for display onto a display device, the 
method comprising the steps of: 

polling a status associated with a stylus pen; 
displaying a matrix, when said status is engaged, 
10 comprising a first plurality of colors, wherein each of said 

plurality of colors corresponds to a location on the display 
device; 

optically reading a particular one of the first 
plurality of colors with said stylus pen and calculating a 

15 location corresponding to the particular one of the plurality 

of colors; and 

displaying a direction box comprising a second plurality 
of colors, wherein each of said second plurality of colors 
corresponds to at least one of a direction associated with 

20 a vector and a position optically reading with said stylus 

pen a particular one of the second plurality of colors and 
calculating at least one of a vector associated with the 
particular one of the second plurality of colors and a 
position of the particular one of the second plurality of 

25 colors. 
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